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Abstract 


The  problem  of  routing  control  in  an  open  queueing  network  under  condi¬ 
tions  of  heavy  traffic  and  finite  (scaled)  buffers  is  dealt  with.  The  operating 
statistics  can  be  state  dependent.  The  sequence  of  seeded  controlled  state  pro¬ 
cesses  converges  to  a  singularly  controUed  reflected  diffusion  (with  the  associated 
costs),  under  broad  conditions.  Due  to  the  nature  of  the  controls,  a  ‘scaling’ 
method  is  introduced  to  get  the  convergence,  since  the  actual  sequence  of  pro¬ 
cesses  does  not  necessarily  converge  in  the  Skorohod  topology.  Owing  to  finite 
buffers,  an  extension  of  the  reflection  mapping  needs  to  be  obtained.  The  op¬ 
timal  value  functions  for  the  physical  processes  converge  to  the  optimal  value 
function  of  the  limit  process,  under  broad  conditions.  Approximations  to  the 
optimal  control  for  the  limit  process  are  obtained,  as  well  as  properties  of  the 
sequence  of  physical  processes.  The  optimal  or  controlled  (but  not  necessarily 
optimal)  limit  process  can  be  used  to  approximate  a  large  variety  of  functionals 
of  the  optimal  or  controlled  (but  not  necessarily  optimal)  physical  processes. 


Key  words:  routing  control,  weak  convergence,  singular  control,  queues  in  heavy- 
traffic,  reflected  controlled  diffusions. 
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Running  head:  Routing  Control 


1.  Introduction 


We  consider  the  problem  of  optimal  or  nearly  optimal  routing  in  a  queueing 
system  under  heavy  traffic  conditions.  The  general  network  model  is  a  “con¬ 
trolled  routing”  form  of  the  general  open  network  dealt  with  by  Reiman  and 
Harrison  [1]  or  Reiman  [2],  where  each  customer  eventually  leaves  the  system. 
We  will  actually  treat  two  special  cases  for  simplicity  in  the  development.  But 
it  should  be  apparent  from  these  cases  that  the  general  open  network  can  be 
treated  in  the  same  way.  The  treated  cases  involve  all  the  bcisic  technique  that 
is  required  for  the  general  case.  In  [2],  one  has  a  finite  set  of  servers,  each  with 
an  infinite  buffer.  We  bound  (and  appropriately  scale)  the  buffers  here.  It  is 
well  known  [1],  [2],  that  under  broad  conditions  on  the  service  and  interarrival 
times,  the  vector  of  queue  length  processes  (with  an  appropriate  amplitude  nor¬ 
malization  and  time  scaling)  converges  weakly  to  a  reflected  diffusion,  as  the 
traffic  intensity  goes  to  unity. 

The  work  in  [2]  required  that  the  system  operating  statistics  not  be  state 
dependent,  and  used  results  for  the  weak  convergence  of  a  sequence  of  sums  of 
mutually  independent  random  variables  to  a  Wiener  process,  together  with  a 
clever  method  to  treat  the  boundary  to  get  the  appropriate  limit.  The  methods 
are  not  extendible  to  the  state  dependent  or  to  the  controlled  case,  where  the 
required  independence  no  longer  holds,  and  the  characterization  of  the  limit 
processes  as  well  as  the  proofs  of  tightness  require  different  methods.  The 
“martingale  type”  methods  for  getting  limit  theorems  for  wide  bandwidth  noise 
driven  systems  seem  to  be  more  appropriate.  In  reference  [4],  there  is  a  study 
of  a  heavy  traffic  problem  under  a  control,  and  the  arrival  and  service  processes 
were  allowed  to  be  state  dependent.  Such  state  dependence  is  natural  for  the 
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controlled  problem,  since  one  might  want  to  let  the  processing  depend  on  what 
is  happening  in  the  system.  In  addition  the  methods  which  are  needed  to 
characterize  the  controls  in  the  limit  problem  as  ‘non  anticipative’,  etc.,  require 
the  use  of  the  same  methods  that  the  state  dependence  requires. 

In  [4],  the  processors  and  the  arrival  sequences  could  be  shut  on  or  off  to 
control  the  flows  and  the  costs.  The  limit  problem  was  an  impulsively  con¬ 
trolled  reflected  diffusion  of  a  non  classical  type,  since  there  was  the  possibihty 
of  multiple  ‘simultaneous’  impulses.  It  was  shown  in  [4]  that  any  sequence  of 
controlled  physical  processes  with  uniformly  bounded  costs  converged  to  a  well 
defined  controlled  limit  process.  Also  the  sequence  of  optimally  controlled  phys¬ 
ical  processes  converged  to  the  optimally  controlled  limit  process,  in  the  sense 
that  the  value  functions  converged.  Also  a  control  which  was  nearly  optimal 
for  the  limit  process  could  be  adapted  to  become  a  nearly  optimal  control  for 
the  physical  process  under  heavy  traffic,  under  quite  broad  conditions.  Such 
results  help  to  justify  the  use  of  heavy  traffic  limit  theorems  for  the  purposes 
of  optimal  other  control.  Because  of  the  behavior  of  the  physical  process  in 
[4]  when  the  on-off  controls  were  used,  the  Skorohod  topology  had  to  be  used 
with  care,  because  the  actual  scaled  queue  length  processes  did  not  converge 
in  the  Skorohod  topology  as  it  is  usually  used.  Also,  that  reference  provided 
convergent  numerical  algorithms. 

In  this  paper,  we  also  deal  with  a  controlled  heavy  traffic  problem.  In  the 
basic  model,  the  routing  of  a  subset  of  the  external  arrivals  could  be  controlled. 
The  aims  are  similar  to  those  in  [4],  The  dynamical  equations  for  the  scaled 
queue  length  process  is  defined.  The  sequence  of  such  processes,  (as  the  traffic 
intensity  tends  to  unity)  might  not  be  tight  in  the  Skorohod  topology,  due  to 
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the  nature  of  the  routing  control.  To  handle  this,  we  start  by  working  with  a 
rescaling  of  the  time,  with  which  we  can  get  tightness,  and  a  characterization 
of  the  weak  limits.  The  rescaling  depends  on  the  control.  After  the  limits  are 
obtained,  an  ‘inverse’  scaling  (dependent  on  the  limit  control)  yields  the  process 
which  actually  characterizes  the  limit  of  the  cost  functionals.  The  limit  process 
is  a  controlled  reflected  diffusion.  But  the  control  is  of  the  ‘singular’  type  in  the 
sense  of  [8].  The  usual  reflection  mapping  which  is  used  to  handle  the  problem 
of  non-negativity  of  the  queue  length  process  must  be  modified  here,  due  to  the 
presence  of  the  finite  buffer.  We  construct  the  proper  reflection  mapping  from 
a  sequence  of  concatenations  of  the  usual  one. 

The  basic  problem  of  interest  is  defined  in  Section  2.  We  work  with  a  system 
of  only  two  processors  for  notational  simplicity.  Also,  until  Section  7,  we  do  not 
have  feedback.  The  addition  of  feedback  is  straightforward,  but  it  seems  to 
be  preferable  to  present  the  ideas  in  as  unencumbered  a  fashion  as  possible. 
The  extension  of  the  result  to  the  general  routing  controlled  open  network  is 
straightforward.  Some  of  the  w'eak  convergence  arguments  and  definitions  from 
reference  [4]  are  used,  but  familiarity  with  that  reference  is  not  necessary.  In 
Section  2,  we  manipulate  the  state  equations  into  the  ‘martingale  plus  drift' 
form  which  will  be  used  in  the  weak  convergence  arguments.  The  reflection 
mapping  result  is  stated  in  Section  3  (and  proved  in  Section  8).  The  required 
rescaling  is  defined  and  the  tightness  and  weak  convergence  proved  in  Section 
3.  We  must  prove  that  the  limit  (singular)  controls  are  non  anticipative  with 
respect  to  the  Wiener  processes  which  ‘drive’  the  limit  process. 

Section  4  is  concerned  with  the  convergence  of  the  cost  functions.  We  prove 
that  there  is  a  routing  control  with  a  uniformly  bounded  cost,  and  show  that  the 
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liminf  of  the  optimal  cost  functions  for  the  physical  processes  is  bounded  below 
by  the  optimal  cost  for  the  limit  process.  In  order  to  show  that  the  limit  of  the 
optimal  costs  for  the  physical  processes  is  the  optimal  cost  for  the  limit  process, 
we  need  to  prove  various  existence  and  approximation  results  for  the  optimal 
policy  for  the  limit  problem.  This  is  done  in  Section  6,  and  uses  the  'limit 
form’  of  the  control  dependent  rescaling  introduced  in  Section  3.  An  interesting 
approach  to  the  approximation  problem  is  discussed.  The  general  rescaling  and 
tightness  methods  are  of  much  wider  use  for  limit  and  approximation  problems 
where  singular  controls  are  involved  and  where  there  might  not  be  convergence 
in  the  Skorohod  topology.  The  developed  approximations  are  then  used  to  prove 
the  approximate  optimality  for  the  physical  processes  of  an  appropriate  nearly 
optimal  policy  for  the  limit  process. 

This  seems  to  be  the  first  work  which  deals  with  such  contjolled  routing 
problems.  Approximations  to  singular  control  problems  for  wide  bandwidth 
noise  driven  systems  were  discussed  in  [6],  but  the  method  used  here  is  rather 
different  and  is  very  natural  for  the  sorts  of  problems  that  are  being  considered. 
Numerical  methods  have  been  developed  for  the  problems  of  this  paper.  The 
proofs  of  their  convergence  require  methods  which  are  similar  to  those  used 
here,  but  since  there  are  many  additional  details,  they  will  be  dealt  with  in  a 
subsequent  paper. 
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2.  Problem  Description 


Until  Section  7,  we  work  with  the  simple  system  of  Figure  1.  This  will  enable 
us  to  develop  the  main  ideas  without  an  excessive  notational  burden.  Also,  for 
notationaJ  convenience,  we  work  with  a  discrete  time  parameter.  The  results 
for  the  analogous  continuous  time  parameter  case  are  the  same.  Each  of  the 
processors  Pq.  Pi  and  Pn  has  its  own  stream  of  arrivals  from  the  exterior.  Pq  is 
used  only  as  an  (instantaneous)  routing  node.  Its  service  time  is  zero.  This  can 
readily  be  changed,  and  the  resi’.ting  network  would  then  be  a  special  case  of 
the  general  network  discussed  u.  Section  7.  The  Pq  routes  to  either  P\  or  Po. 
and  (until  Section  7).  the  completed  services  from  Pj  and  P^  leave  the  system. 
The  routing  decision  is  based  on  the  events  up  to  the  time  of  the  decision.  We 
suppose  that  some  prior  routing  is  assigned  to  each  new  arrival  to  Po,  but  that 
the  routing  nede  can  reassign,  with  an  associated  profit  or  loss.  Next,  we  give 
some  simple  examples. 

Example  1.  There  are  two  classes  of  customers  arrivin^^  (at  random)  at  Pq. 
P,  is  more  efficient  for  class  i,  and  a  prior  assignment  of  class  f  to  P,  is  made. 
But  Pr,  can  reroute  to  the  less  efficient  processor,  depending  on  the  system  state. 
The  cost  of  rerouting  might  be.  for  example,  a  set  up  cost. 

Example  2.  The  case  of  Example  1,  but  with  three  classes  of  customers, 
arriving  at  random.  Class  r  (r  =  1  or  2)  must  be  served  by  Pi.  Class  3  can 
be  served  by  either  processor,  but  one  of  the  Pi  is  more  efficient  (cheaper)  and 
a  prior  assignment  to  that  P,  is  made.  But  Pq  ran  alter  the  assignment.  For 
example,  let  the  P,  represent  data  bases,  with  some  overlap  of  data  files.  A 
subset  of  the  arriving  jobs  need  only  the  'overlap'  data.  But  one  of  the  Pi  is 
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‘faster’  than  the  other. 


Example  3.  Pi  is  cheaper  for  all  customers  arriving  at  Pq.  But,  oue  to 
the  heavy  traffic  conditions,  the  mean  number  of  customers  routed  to  e?  :h  P,  is 
essentially  fixed  (modulo  some  fraction  which  goes  to  zero  as  the  traffic  intensity 
goes  to  unity).  Some  prior  assignment  is  made  but  Po  can  reroute  at  either  a 
■■ost  or  a  savings  if  appropriate. 

In  general,  the  model  can  be  readily  extended  to  handle  rerouting  of  a  cus¬ 
tomer  actually  in  a  queue  as  well  as  reneging. 

In  the  modelling  of  systems  under  heavy  traffic  conditions,  it  has  been  the 
usual  practice  to  suppose  that  the  processors  ‘keep  processing’  and  create  de¬ 
partures  even  if  the  queues  are  empty  [l]-[4].  Whatever  ‘fictitious’  dep-ir'rrr 
occur  due  to  this  convention  are  compensated  for  by  an  added  ‘reflection  term’ 
(our  V  below).  I  hus  each  P;  (t  =  1,2)  has  associated  to  it  a  sequence  of  service 
intervals  which  cover  all  time.  This  convention  simplifies  the  analysis.  Also,  wc 
suppose  (as  is  the  usual  practice)  that  if  a  customer  arrives  at  Pj  or  Pj  when 
the  associated  queue  is  empty,  then  the  service  lime  for  that  customer  is  just 
the  residual  time  of  the  current  service  time  interval  for  that  processor.  As  in 
[l]-[4].  this  convention  does  not  affect  the  limit  processes. 

Since  we  work  in  discrete  time,  it  is  possible  that  multiple  events  can  occur 
at  the  same  time  at  Pj  or  P^.  For  the  sake  of  precision,  we  suppose  that  a 
departure  (real  or  fictitious)  from  a  processor  always  occurs  ‘just  before’  any 
arrival  to  that  processor,  and  that  if  two  arrivals  to  the  same  P,  occur  at  the 
same  time,  then  the  one  from  Po  takes  precedence.  Such  a  conflict  might  arise 
if  there  is  only  space  for  one  customer  left  in  some  buffer,  but  there  are  two 
arrivals.  We  ignore  the^e  distinctions  in  the  notation,  for  simplicity.  It  can  be 
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shown  that  the  precedence  relations  do  not  affect  the  limit. 

Definitions.  We  use  the  notation  of  [4]  whenever  possible,  although  knowl¬ 
edge  of  that  reference  is  not  needed  for  the  reading  of  this  paper.  The  symbol  t 
indexes  the  traffic  intensity;  as  <  —  0,  the  intensity  goes  to  1.  For  each  c  >  0  and 
i  =  1,2,  let  { A),',  n  =  1,2, . . denote  the  sequence  of  service  times  for  P,  and 
let  V  be  the  indicator  function  of  the  event  that  a  service  (real  or  fictitious) 
is  completed  at  P,  at  time  n.  For  i  =  0,1.2  and  each  (  >  0,  let  {aj,',  n  <  oc} 
denote  the  sequence  of  interarrival  times  to  P,,  from  the  exterior  of  the  system, 
and  let  i  =  0.  1.2,  be  the  indicator  of  the  event  that  there  is  an  external 
arrival  to  P,  at  time  n.  Write  t/t  for  [t/e],  the  largest  integer  which  is  no  bigger 
than  t/<.  Define  A'),'  =  >/(  (number  of  customers  waiting  for  or  in  service  at 
P,  at  time  n).  and  set  A"'  '(t)  =  A'’)^.  In  general,  for  a  sequence  {Z^},  define 
the  function  z\i)  =  The  buffer  of  P,,  i  =  1,2,  has  size  BJs/l,  which  we 
assume  is  always  an  integer.  Let  /J,'  denote  the  indicator  of  the  event  that  an 
arrival  at  Po  at  time  n  has  the  pnor  assignment  to  Pi,  and  let  p)/'',  }  ^  t,  be 
the  indicator  of  the  event  that  this  arrival  is  reassigned  to  Pj. 

We  usually  use  the  convention  that  the  superscript  t  is  dropped  whenever 
one  of  the  above  terms  is  used  as  a  summand.  Define  {j  /  i) 

m=0  m=0 

m=0 

n 

msO 

A  '  =  '  -  A’-' 


( 


m=0 

n  n 

C'  =  ^^<Y.  =B.}  +  V^  E  +  ^'rnPi^  "  = 

fn  =  0  m=0 

The  A°’  '  is  the  scaled  total  number  of  arrivals  (by  time  n)  at  Po  which  have 
been  a  priori  assigned  to  P,  (they  might,  of  course,  be  rerouted  by  Fo)- 

The  J^^'‘  are  the  ‘rerouting’  control  terms,  the  scaled  number  of  customers 
originally  destined  for  P,  but  rerouted  to  Pj.  The  is  the  scaled  total  number 
of  'fictitious'  departures  due  to  our  convention  of  continuing  to  ‘process'  even 
if  the  queue  is  empty,  and  L'^  ‘  is  the  number  of  customers  lost  to  F,  when  its 
buffer  is  full. 

The  mass  balanct  equations  can  be  written  as  (discrete  ‘real'  time  and  ‘in¬ 
terpolated’  time,  resp.) 


.V-  =  +  +  + J-'+l-'-t/-',  (2.1) 

A"  '(0  =  A'i '  +  A'  ^t)  +  -  D'  ‘{t)  +  r  ‘{i)  +  y’  '(t)  -  t/'''(t).  (2.2) 


The  cost  fuiictioii.  Let  3  >  0.  c,  >  0.  k,  >  0.  and  let  k{  )  be  a  bounded 
and  continuous  function.  Define  J'  =  (J*^’',  We  use  the  cost  functional 

V‘(r.J‘)  =  £,  f  €-^‘k{X‘{t))dt  +  Er  r  e-^'[kidJ^^'‘{t)  +  kidJ^^  ’{t} 

JQ  Jo 

+  cjdFi’'(f)  +  C2df/^'(/)].  (2.3) 

By  Theorem  7  below,  there  are  routing  policies  J*^’'(  )  for  which 


Define 


sup  y‘(r,  J')  <  oo. 


V‘(x)  =  infV'(z,J'). 


(2.4) 
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The  k{  )  might  be  non-linear.  Such  non-linear  k(  )  occur  when  we  wish  to  model 
the  costs  of  reneging  or  queue  switching,  or  if  we  wish  to  limit  the  possibility  of 
leaving  the  queue  due  to  a  ‘long'  wait.  The  second  term  in  (2.3)  penalizes  the 
overflows  and  rerouting.  One  of  the  ki  can  be  negative  and  we  return  to  this 
case  at  the  end  of  Section  6. 


Definitions  and  heavy  traffic  assumptions.  We  take  many  of  the  defini¬ 
tions  from  [4]  so  that  the  results  of  that  reference  can  be  conveniently  used.  De¬ 
fine  =  Em=i  S'/  =  '  Define  )  ^y  =  max{fm  ; 

tS'^/  £  ^nd  define  S/(  )  analogously.  These  functions  are  the  ‘inverses' 
of  the  functions  Let  denote  the  expectation,  conditioned  on  the 

arrival  and  departure  intervak  which  started  by  S'a'^r,  (except  for  and 

the  control  (routing)  actions  taken  up  to  S^'n.  Define  E'/^  analogously,  where 
S'/„  and  replace  SJ,‘„  and  resp.  Similarly,  define  the  conditional 

variances  var'o;'„,  a  =  a,d.  We  use  the  notation 


^a.n°n  +  l 


=  a 


fl+1  • 


=  A 


rt  +  1 


var-  var^^Al.;,  = 


We  will  use  the  following  assumptions.  A2.1  and  A2.4  are  the  ‘usual’  heavy 
traffic  assumptions.  (A2.4)  basically  says  that  (modulo  a  term  which  goes  to 
zero  as  f  — *  0)  the  mean  rate  of  arrivals  to  P,  equals  the  ‘capacity’  of  P,. 


A2.1.  There  are  real  gai  >  0,  gdi  >  0  and  bounded  and  continuous  real 
valued  functions  a'(  )  and  d'(  )  such  that 


=  9ai+y/(air,  +  o(\/f) 
=  9di  +  V(di„  -f  o(Vf), 
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when 


ai„  =  ..  ),  <f.„  =  ePiXi,,.  ). 

Note  that  A'*  is  the  value  of  the  state  at  the  beginning  of  the  n  +  1st 
interarrival  interval,  and  so  it  is  the  correct  argument  of  the  a’(  )  above,  and 
similarly  for  the  (P(  ). 

A2.2.  1A|;'P,  i,n,£  >  0}  is  uniformly  iniegrahle. 

A2.3.  There  are  p,  sucA  that  =  1  |  a/l  arrival  or  departure  intervals 

starting  by  time  n  and  routing  actions  up  to  time  n  —  1}  =  Pj. 

This  assumption  can  be  weakened  in  many  ways,  allowing  for  batch  rerouting 
and  other  variations,  as  well  as  correlated  routings.  AH  that  is  really  needed  is 
that  'loosely  speaking',  p,  be  a  ‘local  mean’  of  the  conditional  expectations  and 
satisfy  (A2A). 


A2.4.  For  the  p,  defined  in  (A2.3),  Pi^oO  +  9ai  =  9di,  «  =  1,2. 


A2.5.  There  are  continuous  and  bounded  real  valued  functions  <Tat(  ),  <Tdi(  ) 
such  that 


a.n+1 


—  <^o.t(A ^,,,  )  + 


d,n  +  l 


=  )  +  6[ 


where  6,  and  <5'  0  uniformly  in  all  other  variables. 


In  the  sequel,  we  suppose  for  simplicity  that  ail  >  0  for  all  x,  a.  The 

results  are  true  even  if  this  condition  is  violated. 


A  more  convenient  representation  for  A'(  ).  The  2nd  to  4th  terms 
on  the  right  side  of  (2.2)  go  to  infinity  as  c  — ►  0.  For  purposes  of  the  weak 
convergence  analysis,  it  is  helpful  to  center  these  terms  so  that  we  can  work 
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with  martingales  and  processes  of  bounded  variation.  We  follow  closely  the 
procedure  used  in  [4,  Section  3]  with  a  slightly  different  notation.  Access  to 

that  paper  is  not  needed.  Define  the  following  processes. 

•If 

fn=l 

•If 

=  VI  (2.5) 

m  =  l 
«/f 

m=l 

The  summand  in  (2.5)  are  all  centered  about  their  conditional  expectations, 
given  the  ‘past’  data.  Hence,  the  sums  are  martingales.  Henceforth,  we  simply 
write  the  indicator  function  which  appears  in  the  second  sum  as  . 

As  in  [4,  Section  3],  we  can  write  (recall  that  A'’'(t)  =  S’a'(<)/v/f, . . .) 


A''(t)  =  i‘'‘(J/(t))  +  v^  ^  .-=1,2, 


A°''‘(<)  =  A°’''(5°/(<))  +  V7p.  Y.  (2.6) 

mrrl 

D-''(t)  =  D’‘(S’,'(t))  +  V^  Y 

m-\ 

The  first  terms  on  the  right  sides  of  (2.6)  are  just  scaled  martingales.  The  right 
hand  terms  in  (2.6)  ‘blow  up’  as  e  — ♦  0.  In  (2.2),  the  sum  of  the  first  two  minus 
the  3rd  term  of  (2.6)  occurs.  Subtracting  the  far  right-hand  term  on  the  3rd 
line  of  (2.6)  from  the  sum  of  far  right-band  terms  of  the  first  two  lines  of  (2.6), 
and  using  the  heavy  traffic  assumption  (A2.4),  the  expansion  (A2.1),  and  the 
fact  that 

Vi  X!]  A|„  =  ^ -h  0(>/f)  =  Ve  Y  “n' 

m=l  ^  m=l 
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yields  (as  in  [4,  Section  3])  the  expression 

?;■•(«)/£  5^'(<)/t  T/(t)/t 

f  +  «  P.^mflOm-f  ^3  ,  (2.7) 

m=l  m=l  msl 

where  6' ''(■')  is  such  that  sup  li‘’'(<)|  0,  for  each  T  <  oo. 

t<T 

For  i  —  1,2,  let  j4*’'(<),  v4°*'‘(<)  and  denote  the  first  terms  on  the 

right  side  of  (2.6).  Define  bi{x)  =  a*(x)  +  p,a°(a;)  —  d*(x)  and 

B*^(0=  I*  bi{X‘is))ds. 

Jo 

Then,  modulo  an  error  (which  we  absorb  into  ^'’'(•))  of  order  0(f)  due  to  the 
approximation  of  the  sum  by  an  integral,  (2.7)  equals  ^'’'(t)  +  B''‘(t)  and 

=  A'i  +  (i'  ‘(t)  +  i‘^(<)  -  .^>(<)]  + 

+  J'  ‘(<)  +  y‘’‘(t)  -  +  6‘'‘(t).  (2.8) 
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3.  Weak  Convergence 


In  this  section,  we  deal  with  the  weak  convergence  of  the  terms  in  (2.8),  as 
€  — *  0.  Let  D*[0,oo)  denote  the  space  of  i?*^-valued  right  continuous  functions 
with  left-hand  limits  and  C*[0,oo)  the  subspace  of  continuous  functions.  For  all 
the  weak  convergence  work,  we  use  D*[0,oo)  under  the  Skorohod  topology  [5, 
Chapter  3.5].  We  will  often  use  the  Skorohod  representation  [5,  Theorem  3. 1.8] 
so  that  we  can  always  assume  that  if  a  sequence  of  processes  converges  weakly, 
then  the  convergence  is  (w.p.l)  also  pathwise  in  the  topolog>’  of  the  path  space. 

There  are  two  main  problems.  First,  little  is  known  about  the  control  terms 
In  general,  even  if  bounded,  they  need  not  converge  in  the  Skorohod 
topology.  Indeed,  their  behavior  can  be  quite  ‘wild’.  The  pseudopath  topology 
[7]  could  be  used,  as  it  has  been  in  [6]  for  some  approximation  and  convergence 
questions  arising  from  systems  with  wide  bandwidth  noise  disturbances  under 
singular  controls.  For  our  purposes,  it  is  more  convenient  to  work  directly  with 
the  Skorohod  topology,  but  with  a  rescaled  set  of  processes.  (Some  comments 
on  the  relations  between  scaling  and  the  pseudopath  topology  are  in  [9].)  After 
getting  the  desired  weak  convergence,  we  invert  the  ‘limit’  of  the  rescaiings  to 
get  the  result  for  (2.8). 

The  second  problem  concerns  the  treatment  of  the  reflection  terms 
and  !/'’'(  •).  Owing  to  the  presence  of  the  upper  boundary,  the  reflection  map¬ 
ping  theorem  of  [1],  [2]  cannot  be  used  directly.  The  following  extension  is 
proved  in  Section  8. 

Theorem  1.  Lei  Q  be  a  k  x  k  probabUiiy  transition  matrix  whose  spectral 
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radius  is  less  than  unity.  Let  2(-)  €  Z?*[0,oo)  and  consider  the  equation: 

x{t)  =  z{i)  +  iI-Q')y{t)-u{t).  (3,1) 

There  is  a  continuous  function  (in  the  topology  of  uniform  convergence  on 
bounded  time  intervals)  F{  )  suck  that  (y(-)i  “(•))  =  follow¬ 

ing  properties:  F(-)  maps  C*[0,oo)  into  C*'[0,oo)  and  D*[0,oc)  into  D*[0,oo); 
for  i  —  1,2,  i/'(  )  and  u'(  )  are  non-decreasing  and  increase  only  when  x'{t)  =  0 
and  x'{i)  =  B,,  resp.  Eqn.  (3.1)  holds  and  x*(<)  6  [0,B']. 

Using  the  martingale  properties  of  the  sums  defined  in  (2.5),  it  is  not  hard 
to  prove  Theorem  2.  In  fact,  the  proof  of  the  first  paragraph  is  given  in  Lemma 
5.2  in  [4].  and  the  proof  of  the  second  paragraph  is  in  [4,  Theorem  5.1]. 

Theorem  2.  .4ssume  (A2.1)  to  (A2.5).  Then,  for  a  =  a  or  d,  the  processes 
with  values  converge  weakly  to  the  deterministic  functions 

with  values  t/gd  and  tgai,  resp.  The  processes 

{A'''(),i'‘().(i°i'‘(-).A'’2’'()).^'''(-).£>"’'(’),e>0} 

are  tight  and  the  limit  of  any  weakly  convergent  subsequence  of  the  five  sequences 
(we  always  pair  together  A°^’‘(  )  and  A°^’'(-))  are  orthogonal  continuous  mar¬ 
tingales. 

The  quadratic  variations  of  the  limit  martingales  are,  resp.,  the  weak  limits 

of 

I  Ea<(A-'(s))ds.  I  =  1,2,0, 

Jo 

l\i,{X\8))ds,  1  =  1,2,  (3.2) 

Jo 

where 
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^ao(^ )  —  9a0 


Pl(l_-_Pl)  _  -P1P2 
-P1P2  P2(l  -  P2) 


+  ffa0«^a0(2;) 


1 

.  P1P2 


^M=91<tI{x),  .=  1,2. 


P1P2 

1 


Since  the  proofs  of  an  almost  identical  result  is  in  the  cited  reference,  we 
omit  it  and  comment  only  on  how  (3.2)  is  calculated  in  one  case. 

The  quadratic  variation  of  the  discrete  parameter  martingale  j4o’'(  )  is  (re¬ 
calling  that  the  argument  of  <Ta,(  )  is  the  state  at  the  time  of  arrival  of  the  mth 
customer) 

</(  t/c 


m  =  l 


-f  (small  terms). 


Neglecting  the  small  terms  (which  go  to  zero,  as  c 
quadratic  %’ariation  of  ,4'''(-)  as 


f 


t,C 


msl 


0),  we  can  write  the 


+  ,  ^  (3.3) 

Ct  ’ 

m=l 

The  variance  of  the  second  term  in  (3.3)  is  0(f<)  due  to  the  centering  of  the 
summands  about  the  conditional  expectations.  The  first  term  in  (3.3)  can  be 
written  as  (modulo  an  error  of  order  0{y/()) 

f  9li<TlAX^{s))ds.  (3.4) 

Jo 

Thus,  we  obtain  the  first  line  of  (3.2),  for  .  =  1,2. 


The  time  rescaling.  The  weak  convergence  proofs  for  the  terms  in  (2.8) 
is  facilitated  by  means  of  a  rescaling  or  ‘stretching  out’  of  time.  Define  T'(  •)  by 

^(nf)  =  nc  +  ^  [pi?  +  pU  - 

m=l 
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and  for  t  €  (ne,  ne +  €),  define  T‘(t)  to  be  the  piecewise  linear  interpolation.  Let 
T’‘(  )  denote  the  inverse  function  to  T^(  )-  For  any  function  <^(-)  on  [O.oo),  define 
the  function  (^'(■)  by  <^'(<)  =  4>{T‘(t)).  Similarly,  define  =  A“''(T’'(<)), 

etc. 

Theorem  3.  Assume  (A2.1)  to  (A2.5).  Then 

>  0}  (3.5) 

IS  Ughi  and  all  Itmtis  are  continuous  processes.  Also 

>  0}  (3.6) 

IS  tight  and  the  limits  of  any  weakly  convergent  subsequence  of  the  set  of  five  se¬ 
quences  are  orthogonal  continuous  martingales.  Let  e  index  a  weakly  convergent 
subsequence  of  (3.5),  (3.6),  and  denote  the  limits  by  the  same  letters,  but  with 
the  e  dropped.  Then 

A'‘(0  =  A'‘(0)  +  B'(i)+  [.4'(0  +  A°’{t)  -  £»•(<)]  +  Y'(t)  -  U‘{t)  +  P\t)  -  r^{t). 

(3.7) 

y''(  )  increases  only  when  A'’(t)  =  0  and  U'{  )  increases  only  when  A''(t)  =  B,. 
Also. 

B'{t)  =  f  bi(X{s))dt{s).  (3.8) 

Jo 

The  quadratic  variations  of  the  martingales  are 

/‘ E<,.(A(s))dr(s),  I  =  0,1,2, 

Jo 

Zii(X{s))df{s),  i=l,2.  (3.9) 
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For  the  particular  chosen  weakly  convergent  subsequence,  let  T\  denote  the 
minimal  a-algebra  which  measures  {P(s),s  <  l},  where 


P{s)  =  {Xis).J^\s),P\s),A\s),A°\s),D'{s),Tis),  i  =  1,2). 


Then  the  martingales  are  all  Ti-martingales. 

Proof.  The  set  (3.6)  is  tight  and  has  the  asserted  properties  by  Theorem  2, 
since  (3.6)  is  just  the  sequence  dealt  with  in  Theorem  2,  but  with  a  ‘stretched 
out'  time  scale.  The  {T'(  ),  J*^''(  ),  J^'''(  ),  f  >  0}  are  tight  since  their  incre¬ 
ments  between  any  i,t  +  s  are  bounded  by  s  +  0{y/€).  The  set  {B‘{  ),  f  >  0}  is 
obviously  tight. 

To  treat  the  V‘  '(•)  and  we  use  the  representation  of  the  reflecting 

terms  of  Theorem  1.  Thus,  there  is  a  continuous  function  (in  the  sense  of 
Theorem  1)  Fo(  )  such  that 


(V'(  ).i-'(  ))  =  fo(A-o‘..4‘'‘(  ),i°‘''(-),I)‘'‘(-)-5’'‘(-)../''''(  ),^'''''(  ),  »•=  1.2). 

The  tightness  of  {V',  r'(  ),  -V‘(  ),  e  >  0}  and  the  continuity  of  the  weak  limits 
follows  from  this  and  the  fact  that  the  argument  processes  of  fo(  )  tight  and 
have  continuous  weak  limits.  Also,  the  properties  asserted  below  (3.7)  hold. 
The  representation  (3.8)  follows  from  the  equality 
ff’U)  ft 

B-'‘(t)=  /  bi{X^{s))ds=  /  bi(X‘(T‘{s))dT‘{s),  (3.10) 

Jo  Jo 

as  we  will  now  see.  Abusing  notation,  let  t  index  a  weakly  convergent  subse¬ 
quence  of  the  sets  in  (3.5),  (3.6),  and  suppose  that  the  Skorohod  representation 
is  used  so  that  we  can  assume  that  ail  weak  convergences  are  convergences  w.p.l 
and  are  uniform  on  each  bounded  time  interval  (since  the  limit  processes  are 
continuous  w.p.l).  Since  the  T’‘(  )  satisfy  |T’'(<-l-«)  — T’'(t)l  =  0(s),  the  uniform 
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convergence  (on  each  [0,/])  of  T’'(  )  to  continuous  f{  )  and  X'(  )  to  continu¬ 
ous  X(  )  and  (3.10)  yield  the  assertion.  A  similar  proof  yields  the  analogous 
assertion  for  the  quadratic  variation  terms. 

The  last  sentence  of  the  theorem  is  proved  in  the  same  way  that  (5.4)  in 
[4]  is  proved,  via  use  of  the  ‘martingale  method’,  and  we  only  do  one  case. 
Let  /»(•)  be  an  arbitrary  real  valued,  bounded  and  continuous  function  of  its 
arguments  and  for  arbitrary  n,  let  <  t  <  t  -I-  s,  t  <  n.  Define  P'(t)  =  (A'‘(<), 
/12,.(<)  '(<),  !)*■'(<),  i  =  1,2).  Let  e  index  a  weakly 

convergent  subsequence  of  {.P'(  ),  (  >  0).  It  can  be  shown  that 

EhiP‘(t,),i  <  n)[.4‘  '(f  +  s)  -  A‘  '(t)]  =  0. 

This  last  expression  can  be  shown  either  by  the  ideas  leading  to  (5.4)  in  [4].  or 

by  a  direct  calculation  using  the  definition  of  the  conditional  expectation  n 

and  the  fact  that  the  summands  in  A'  '(•)  are  centered  about  their  conditional 

expectations,  given  the  ‘past’.  By  the  weak  convergence  and  the  fact  that 

sup  E[A'  '(t)]^  <  oc  for  each  /  <  oc.  we  have 
€ 

Eh{P{U),  i  <  n)[A'(t  +  s)  -  A’(<))  =  0. 

The  arbitrariness  of  /j(  ),  t,.  n.  t.  t  +  s,  implies  that 

E[A(<  +  s)  -  A(<)  I  P(u),  u  <  t]  =  0, 

which  yields  the  assertion.  Q.E.D. 

The  inversion  of  T{  ).  We  next  deal  with  the  inversion  of  the  time  rescaling 
T(  ),  to  get  the  appropriate  ‘limits’  of  the  original  sets  of  processes  in  (2.8). 
Whether  this  ‘inversion’  can  be  done  or  not  depends  on  the  controls.  Clearly  if 
ail  arrivals  at  Pq  are  rerouted,  then  for  each  t  >  0,  T‘{t)  — »  oo  as  e  — »  0  and 
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T{t)  =  0  and  no  inversion  is  possible.  However,  since  the  costs  associated  with 
this  policy  go  to  infinity  as  e  — >  0,  such  causes  can  be  excluded.  It  will  turn  out 
that  for  the  controls  of  practical  interest,  the  inversion  can  be  done. 

Lemma  4.  .4ssurne  (A2.1)-(A2.5)  and  ihai 

</' 

sup <  oo  (3.11) 

^  m  =  1 

for  each  /  <  oc .  Then  f(t)  <  oc  w.p.l  for  each  i  <  oc  and  T{t)  — ►  tc  w.p.l,  as 
t  —  oc. 

The  proof  is  easy  and  is  omitted. 

For  each  t  >  0,  define  the  random  variable 

T{t)  =  min{r  ;  t(T)  =  t). 

The  set  {T(f).  s  <  x }  are  E, -stopping  times,  since  {T(<)  <  t)  =  {T(t)  <i]£ 
Tr  for  all  r.  Define  the  cr-algebras  Tt  —  For  any  process  o(  ),  define 

the  rescaled  process  <f>(  )  by  o(0  =  except  let  A°{  )  and  D°{  )  denote 

.4"’(T(  ))  and  b^(T{  )).  resp.  Then  i.s  the  minimal  (r-algebra  induced  by 
{P(s),s<<}  =  {P(T(s)),s<t}. 

Theorem  5.  Assume  (A2.1  )-(A2.5)  and  (3.11).  Then 

X'H)  =  A’(0)  -F  B'{t}  +  [.-Vin  +  A°'(t)  -  D'(()]  +  Y'{t) 

-  +  J'bi)  (3.12) 

The  y'(  )  and  [/'(■)  increase  only  when  X'{i)  =  0  (X'(<)  =  Bi,  resp.).  The 
martingales  are  all  Tt-maritngales.  The  quadratic  variations  are  given  by  (3.9) 
with  T{t)  replaced  by  t  and  X(  )  by  X{  )■ 
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The  prr  -f  is  just  a  consequenre  of  Theorem  3,  Lemma  4  and  the  properties 
of  the  T(t).  The  details  are  c>mitted. 

Remarks  on  the  representation  of  the  martingales.  Since  the  five 
processes  .4'(  ),  D'(-),  i  =  1,2,  and  (j4°'(  ),  are  mutually  orthogonal 

martingales,  we  can  represent  them  as  stochastic  integrals  with  respect  to  mu¬ 
tually  independent  Wiener  processes  Ua,(  ),  tUd,(  ).  If  the  ffaff  are  never  zero 
(which  we  ha\e  assumed  for  convenience  in  this  paper),  then  the  u'o,(  )  are  all 
,r,-\N'iener  processes  Otherwise,  we  need  to  augment  the  probability  space  and 
filtration  by  adding  Wiener  processes  which  are  independent  of  all  the  processes 
originall)'  defined  on  the  probability  space.  We  can  write  the  martingales  in  the 
form 

A'[»  =  9l!'  f  ^a,(X{-^))dUa,(s)=  f  i:'f(X{s))dUai{s),i=l.2. 

Jl  Jo 

D'it]  =  j‘  ^,(X(s)}du^,(s)  = 

(  )  =  I -‘>oiX(s))dwao(s). 

If  the  {/’-'()../*''()•<  >0)  is  tight,  then  the  time  change  1  t’(t)  '\s  not 

needed,  and  one  can  work  directly  with  the  original  processes  A''(  ) .  We  will 

next  giv-'  a  result  for  this  case  which  will  be  useful  below.  First,  we  define  some 
new  process  by  a  n  Drmalization  of  l-ie  summands  in  the  expressions  j4o 
yio’  '(5g  *(t))  and  appearing  in  (2.6).  These  new  processes  will 

actually  converge  weakly  to  the  Wiener  processes  Wap(  )  'T)cme 
s;'i<)/< 

m  =  1 

S'/tt}/' 

IU(0  =  V^ 

m=  1 


)]■  (l-Ai„/A’„),  .  =  1,2, 


)J  (1-q‘„/q’„),  .=  1,2, 


f  Y:li\x{s))dw„{s). 

Jo 
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.vy-)  =  v^“E  flKl/l  }■ 

Theorem  6.  Assume  (A2.1)  to  (A2.5)  and  suppose  ihat  ), 

f  >  0}  IS  tighi.  Then  {note  that  we  patr  the  two  components  of  Hao(  )) 


{A'M  alii} 


IS  Ughi.  Let  e  index  a  weakly  convergent  subsequence  and  denote  the  limits 
by  the  same  letters,  but  uiihoui  the  e  superscript.  Let  be  the  minimal  <r- 
algebra  which  measures  the  limit  process  for  s  <  t.  Then  the  Wa(  )  are  mutually 
independent  standard  Tf  Wiener  processes,  and 

-4'(0=  f  Zli\X(s))d\V,,{s), 

Jo 

-4°(0=  f  '^ii\X(s))d\VMs),  (3.15) 

Jo 

D'(t)=  f  Zli'^(X(s))d\VMs)■ 

Jo 

Also  (3.12)  holds. 

Proof.  It  is  easy  to  sliow  the  ‘Wiener  process’  result,  owing  to  the  center¬ 
ing  of  the  summands  and  the  normalization  by  the  inverse  square  root  of  the 
covariance.  The  rest  is  as  for  Theorem  3,  except  for  the  representation  (3.15). 
This  can  be  obtained  by  using  the  tightness  and  a  discrete  time  approximation, 
and  the  details  are  omitted.  Q.E.D. 
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4.  Boundedness  and  Approximation  to 


First,  we  show  that  there  is  a  control  for  which  the  costs  are  uniformly 
bounded. 

Theorem  7.  Assume  (A2.1)  to  (A2.5),  and  let  V‘{x,0)  denote  the  cost 
when  J^^-^it)  =  =  0.  Then 

sup  Fr  V'(if.O)  <  OO. 

i,T 

Proof.  It  is  enough  to  prove  that 

sup  E[U*'‘(n  +  1)  -  (/•■'(n)]  <  oo. 

e,n,t 

Define  M'  ‘(t)  =  [.4'  '(/)  +  -  £>’■'(<)]•  We  let  i  =  1,  since  the  proof  is 

the  same  for  i  =  2.  For  an  integer  n,  define  the  stopping  times  (omitting  the  n 
and  (-dependence  in  the  notation) 

Ti  =  min{<  >  n  :  =  Bi), 

T:2rn  =  min{t  >  rjm-i  ;  <  Bi/2)  n  (n  -I-  1), 

r-jm  +  i  =  min{/  >  T2m  ■  =  Bi}  n  (n  -P  1), 

Define  .V^  =  min{m  :  r^m  =  n  -f-  1).  Recall  that  U^  ‘{  )  can  increase  only  on 

the  intervals  [r2m-i,T2m]  and  not  on  (t2„,,  T2rr,+}).  Then 

n:+i 

+  1)  -  [(A"’'(r2„)  -  A'  '(r2^_i)) 

m=l 

-  (A/>''(r2„)-Af>’'(r2„-,))-(B*’‘(r2„)-B'''(r2m-i))].  (4.1) 

By  (4.1)  and  the  square  integrable  martingale  property  of  Af'’'(  )  and  the  Lip- 
schitz  continuity  property  of  B*’'(-),  there  is  a  constant  Ko  such  that 

ElU''‘(n  -b  1)  -  t/'''(n)|  <  Ro  +  E(N‘  +  l)Bi  (4.2) 
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Thus,  to  prove  the  theorem,  we  only  need  bound  EN^,  uniformly  in  n  and  f. 

Given  Qq  >  0,  there  is  6o  >  0  such  that  for  all  bounded  stopping  times  and 
for  small  t 

P\  sup  sup[B’'(s)  +  M'’'(s)-(B^'(r)  + Af^''(r))] 

Lt+4o>»>t 

>  —  I  data  up  to  t|  <  1  —  oq-  (4  3) 

This  implies  that 

P{T2m  -  Tsm-I  >  ^0  [data  up  to  time  r2m-i}  >  oo-  (4.4) 

Consider  the  problem  of  a  sequence  of  ‘Bernoulli’  trials,  where  the  conditional 
probability  of  success,  given  the  past  data,  is  >  Qq  and  on  each  success  ‘time’ 
advances  by  Sq.  An  upper  bound  for  our  EN^  is  just  the  mean  number  of  trials 
that  are  needed  to  have  l/6o  =  ni  (the  next  largest  integer)  successes.  Since 
the  mean  number  of  required  trials  is  monotonic  in  the  (conditional)  probability 
of  success,  we  get  an  upper  bound  by  assuming  that  (4.4)  is  an  equality.  Then 

P{k  trials  needed}  =  ^  ^  ~ 

which  implies  that  all  moments  of  A’4  are  bounded,  uniformly  in  n  and  e. 
Q.E.D. 

We  remark  that  the  proof  and  the  uniform  square  integrability  of  the  incre¬ 
ments  in  M‘{  )  and  B‘(  )  (on  unit  intervals)  implies  that 

sup  £|6”''(n  1)  -  (/•■'(n)|2  <  oo.  (4.5) 

t,n 

The  following  corollary  will  be  useful  later.  It  is  just  a  consequence  of  Theo¬ 
rem  7,  the  structure  of  the  cost  and  the  discounting.  Define  ^‘(r)  =  inf  V"(j;,  J') 
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Corollary  8.  y455ame  (A2.1)-(A2.5).  Given  ^  >  0,  there  are  To>  Q  and  a 
family  of  6-opiimal  controls  .//(•)  such  that  J/(  )  do  not  change  after  time  Tq 
(re.,  after  To,  there  is  no  rerouting). 

A  very  similar  proof  to  that  of  Theorem  7  yields  the  following; 

Theorem  9.  Assume  (A2.1)  to  (A2.5).  If 

sup  E[J'^’%t  +  T)  -  J‘^'(T)]  <  oo,  i  ^  ji,  «  =  1,2.  (4.6) 

£ 

Then 

sup  ::[i/'  '(<  +  T)  -  W'^iT)]  <  oo. 

€ 

If 

{J'^''(<  +  D  -  J’^''(T),e  >  0},.-  ^  j,  i  =  1,2,  (4.7) 

IS  uniformly  integrable  for  each  t,  then  so  is  {U'’‘{t  +  T)  —  U''‘{T),  (  >  0), 

i=  1,2. 
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5.  The  Limit  Control  Problem 


Definition.  J(  )  =  is  said  to  be  an  admissible  control  for  the 

limit  controlled  reflected  diffusion  (3.12)  if  it  is  non-anticipative  with  respect  to 
the  set  of  Wiener  processes  U’(  )  =  Wdi{  },  i  =  1,2,  ii;oo(  ))  which  ‘drive’ 

the  martingales  (^*(  )>  -^(  li  •  =  i.2)  (see  the  representation  (3.13)), 

and  satisfies  ^“(0)  =  0  and  J°(  )  is  non-decreasing,  for  a  =  12  or  21.  W’e  often 
say  simply  that  the  pair  (J'(  ),  1V(-))  is  admissible.  The  cost  functional  for  the 
limit  problem  is 

^oc  *co 

V(x,J,W)  =  E,  e-‘^‘k{X(t))dt  +  Er  e-^'[kidJ^\t)  +  k:,dJ^\t) 

Jq  Jo 

+  cidV\t)  +  C2dU\l)].  (5.1) 

The  \V(  )  appears  in  (5.1)  as  well  as  J(  ),  since  the  value  of  the  cost  function 
will  depend  on  the  joint  distribution  of  (J(  ),  H’(  ))- 

Theorem  10.  Assume  (A2.1)-(A2.5)  and  that  for  each  n 

sup  £[(J‘2  +  1)  _  Ji2.‘(n))  (J2i.‘(„  +  1)  _  J2i  '(n))]  <  oc.  (5.2) 

€,n,i 

Let  e  index  a  weakly  convergent  subsequence  of  (3.5),  (3.6)  with  limit  denoted 
by  (T(  }, . . .).  Let  the  retransformed  processes  defined  above  and  in  Theorem  5 
be  denoted  by  (T(-) _ ).  Then 

lim<V'(s;.J‘)>  (5.3) 

where  W(  )  =  (tra,(  ),u/di(  ),i  =  1,2)  is  the  Wiener  process  which  is  used  to 
represent  the  martingales  (see  (3.13)).  If 

1)  -  7'2  ‘(rj),  J2‘'‘(n  -)-  1)  -  /^'’'(r.),  e  >  0,  n  <  oo}  (5.4) 
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:s  uniformly  tniegrabk,  then 


v^{x,r)-^v(x,j,w). 


(5.5) 


Proof.  The  hypothesis  (5.2)  implies  that  inf  £T'(<)  -+  oo  and  Ef{t)  — ►  oo 
as  <  — ►  oc  .  Thus,  the  ‘ins’erse’  transformation  T(-)  is  well  defined.  It  also  implies 
that  we  need  only  work  on  a  finite  interval  (see  Corollary  8).  For  simplicity,  we 
work  with  only  a  couple  of  the  terms  of  the  cost  functional.  We  have 


/  e-^"'k{X‘it))dt=  I  e-^^‘(‘)it(X'(l))<ir(/), 
Jq  Jo 

f 


'\t)=  f  (5.6) 

Jo 

By  the  weak  convergence  and  the  argument  of  Theorem  3,  the  right  sides  of 
(5.6)  converge  in  distribution  to  the  left  sides  of 

r  e-^^''U-(A'(<))rfr(0  =  r  e-^‘/fc(A'(0)d<, 

Jo  Jo 

r  =  re-^'dJ^Ht)-  (5.7) 

Jo  Jo 

The  left  sides  of  (5.7)  equal  the  right  sides  of  (5.7)  by  the  rescaling.  The  the¬ 
orem  follows  from  the  cited  convergences  (together  with  those  for  the  other 
components  of  the  cost)  and  Fatous’  Lemma.  Q.E.D. 


Theorems  3  and  5  imply  that  every  limit  of  a  weakly  convergent  subsequence 
is  a  legitimate  control  problem  in  the  sense  that  the  pair  (J(  ),  W(  ))  which 
occurs  in  the  representation  of  the  limit  is  admissible.  This  fact  and  Theorem 
10  imply  the  following. 

Theorem  11.  Assume  (A2.1)-(A2.5).  Let  J*^''(  ),  )  denote  the  op¬ 

timal  controls  for  the  physical  process.  Define 


V"(x)  =  infK‘(z,J'), 


26 


Then 


lim,V‘(x)  >  V(x).  (5.8) 

Remark.  We  note  that  (5.2)  can  be  assumed  in  Theorem  11.  If  it  doesn’t 
hold  for  the  optimal  policy,  for  each  6  >  0  it  will  hold  for  the  6-optimal  policy, 
owing  to  the  discounting  and  Corollary  6.  We  want  to  prove  that  (5.8)  is  an 
equality.  To  get  the  equality,  we  will  need  to  use  the  fact  that  V’'(x)  is  actually 
an  optimal  cost.  In  order  to  do  this,  we  need  first  to  study  approximations  to 
the  control  problem  for  the  limit  model  (3.12),  (5.1).  We  will  show  that  there  is 
an  optimal  policy  for  the  limit,  and  that  it  can  be  approximated  by  a  policy  that 
we  can  apply  to  the  A''(  )  process,  and  which  will  be  ‘recovered’  under  the  weak 
convergence.  Such  results  will  get  us  the  desired  equality  in  (5.8)  (Theorem 
17),  together  with  a  basis  for  an  effective  computational  approximation.  The 
computational  methods  and  associated  proofs  will  be  dealt  with  in  a  subsequent 
paper. 
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6.  Approximations  for  the  Limit  Problem,  and  Convergence  of  the 

Costs 


In  order  to  prove  equality  in  (5.8),  we  first  establish  the  existence  of  an 
optimal  policy  for  (3.12),  (5.1),  and  then  obtain  a  sequence  of  approximations 
to  the  optimal  control.  We  will  use  the  following  assumption. 

A6.1.  /!:(•),  6,(  ),  <Tia(  ),  )  continuous. 

Theorem  12.  Assume  (A6.1).  Consider  the  limit  control  problem  (3.12), 
(5.1).  There  is  an  optimal  policy  J(  )  in  the  sense  that  there  is  (A(  ),  J(  ), 
Vr(  ),...)  satisfying  (3.12),  where  W(-)  =  {wa«(-)i  ^di{  ),  i  =  1,2}  ‘drives’  the 
martingales  .4'(  ), _  as  in  (3.13)  and  the  pair  (J(  ),  W(  ))  is  admissible  and 

V{xJ,W)<V(x,J,W), 
for  all  admissible  pairs  (J(),  lf’(  ))- 

Proof.  The  proof  is  very  similar  to  those  of  Theorems  3  and  5,  and  we  make 
only  a  few  comments.  Let  (J"(-)>  M^"('))  he  an  admissible  pair  for  (3.12),  and 
write  the  corresponding  form  of  (3.12)  as  (ji  /  i) 

A'  "(0  =  A'j  +  S'  "(0  +  [A'’"(t)  +  -  £»^'"(t)] 

+  [J'^’^it)  -  J’'-^{t)]  +  y‘’"(t)  -  t/’’"(0,  (6.1) 

where  5''"(t)  =  f*  b,{X‘-'‘{s))ds  and  W"(.)  =  «(•),  <(■),  i  =  1,2}  ‘drives’ 
the  martingales  A''"(),...,  as  in  (3.13).  Let  (,/"(•),  W"())  be  a  minimizing 
sequence  in  that  V{x,J",  W”)  }  V(x). 

By  Theorems  7  and  10,  V(x)  <  oo.  Hence 

sup  f:[J*^-"(t)  +  7*'’"(1)]  <  oo, 
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for  each  t  <  oo.  Define  the  time  change 

r"(f)  =  f  +  + 

and  the  inverse  7^(<)  =  min{r  ;  T"(r)  =  t}.  Analogous  to  the  notation  used  in 
Theorem  3,  define  A'"(  )  =  A'" (f"( Then 

=  A'j  +  +  [A‘’"(f)  +  -  D‘>(f)] 

+  -  P'  ”{t)  +  -  [/■■"(/). 

As  in  Theorem  3,  there  is  a  function  Fo(-)  which  maps  C*[0,oo)  into  C*[0,oo), 
for  the  appropriate  integer  k  and  is  continuous  in  the  topology  of  uniform  con¬ 
vergence  on  bounded  time  intervals  and  is  such  that  for  all  n 

The  set  {t"(  ),A'"(  ),  >(•),  £)•'"(•),  i=  1,2,  n  <  oo} 

is  tight.  Abusing  notation,  let  n  index  a  weakly  convergent  subsequence  w'ith 
limit  denoted  by  (r(  ),  A'(  ),...).  As  in  Theorem  3  the  (A’(  ),  A^(  ),  (A’°(  ), 
j4^°(  )),  D'(  ),  D^(  ))  are  orthogonal  continuous  martingales  w’ith  quadratic 
variation  defined  by  (3.9).  Define  the  inverse  scaling  T{t)  =  min{r  :  T'(r)  =  f), 
and  the  rescaled  processes  A'(<)  =  A'(T(<)), . . ..  Then  (3.12)  holds,  and  the 
martingales  have  the  representation  (3.13)  with  respect  to  some  Wiener  process 
W(  )  =  (u;a,(  ),  Wdi{  ),  i  =  1,2)  such  that  the  pair  (^(O,  W(  ))  is  admissible. 
By  an  argument  which  is  almost  identical  to  that  of  Theorem  10,  we  have 

\mr,V{x,r,W)>V{x,J,W).  (6.2) 

We  must  have  equality  in  (6.2)  since  F(r,  J",  W")  )  V{x).  Thus,  (/(■),  H’(  )) 
is  an  optimal  admissible  pair.  Q.E.D. 
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integrable,  and  so  is  {(/‘'^'(n  +  1)  —  U*'^(n),  n  <  oo,  iV  <  oo,  i  =  1,2}  by 
Theorem  13.  Since  T  oo.  we  can  suppose  that 

pathwise.  The  theorem  follows  from  this  convergence,  the  cited  uniform  inte- 
grability  and  an  argument  similar  to  that  in  Theorem  10.  Q.E.D. 

Definition.  A  solution  A'(  )  to  (3.12),  (3.13)  is  said  to  be  unique  in  the  weak 
sense  if  tlie  distribution  of  the  admissible  pair  (./(•),  1^(  ))  determines  that  of 

In  order  to  obtain  our  approximation  results  we  require  that  for  each  6  >  0, 
there  is  a  (f-optimal  control  which  gives  a  well  defined  solution  of  (3.12),  (3.13). 

A6.2.  For  each  6  >  0,  there  is  a  6-optimal  control  J()  for  which  (J().  H’()) 
IS  admissible  for  some  ir(  )  and  the  corresponding  solution  A'(  )  to  (3.12),  (3.13) 
IS  unique  in  the  weak  sense. 

In  the  next  theorem,  we  show  that  there  is  a  6-optimal  control  which  is 
bounded,  piecewise  constant,  and  jumps  ‘in  increments’. 

Theorem  15.  Assume  (A6.1)  to  (A6.2)  and  lei  (J(  ),  1V(  ))  be  a  6-optimal 
pair  satisfying  (AC. 2),  with  A'(  )  denoting  the  corresponding  solution  process. 
For  A  >  0  and  p  >  0,  define  the  control  J^p(  )  as  the  piecewise  constant  control 
satisfying:  dJ’^^(t)  =  0  on  the  interval  (nA,nA-(-  A)  and  on  [0,  A).  For  ifc  >  0 
and  n  >  1,  set  dJ^^inA)  =  kp  if  J‘^{nA)  -  J’^{nA  -  A)  €  [kp,  kp  -k  p).  Then 

limV(i,JAp.VV)  =  K(x,J,W).  (6.4) 

^,P 

Proof.  By  Theorem  13,  we  can  suppose  that  J{  )  is  uniformly  bounded. 
By  construction,  (Jai>(  )>  ^(O)  *8  an  admissible  pair.  A  solution  to  (3.12), 
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The  following  lemma,  whose  proof  is  similar  to  that  of  Theorems  7  and  9, 
will  be  useful  later. 

Theorem  13.  Assume  (A6.1),  and  let  (Jn(  ),  W^n(-))  admissible,  with 
■^n(  ).  TnC  )  and  (/„(■),  be  the  associated  state  and  reflection  process.  If 

+T)  -  Ji:{T),n<  oo],j  ^  i,  j  =  1,2, 

IS  uniformly  iniegrable,  then  so  is 

{ll(t+T)-r^{T),n<cx}.jgii,  j  =  1,2. 

Theorer'  14.  Assume  (A6.1)  and,  for  small  6  >  0  let  (Jo(  ),  Wo(  ))  he 
a  6-opttmal  admissible  pair,  with  A'o(  ),  being  the  associated  solution  to  (3.12). 
Define  tv  =  sup{t  ;  Jq^U]  <  N,  Jo’(<)  <  A"),  and  let  J^(-)  be  the  policy  which 
equals  Jo(  )  until  ts  ,  and  is  constant  thereafter.  Vl'nie  the  solution  to  (3.12)  os 

A'’'^'(0  =  A‘(0)  +  B'-'^(r)  +  [A'-^(t)  +  A°''^(t)  -  b'-^’(t)] 

+  Y'’^\t)  -  U'-^'{t)  +  J^‘-^'(0  -  J'^-"''{t). 

Lei  lVj^(  )  be  the  set  of  Wiener  processes  which  ‘drives  '  the  martingales  (A''^  (•). 
. . .).  Then,  as  N  — »  oc, 

V(x,J^',\Vi^')^V(x,Jo,Wo).  (6.3) 

Proof.  We  can  suppose  w.l.o.g.  that  there  is  a  T  <  oo  such  that  Jo(  ) 
is  constant  after  T  (by  an  argument  similar  to  that  leading  to  Corollary  8). 
Since  D'^iT)  ]  Jl{T),  and  EJi{T)  <  oo,  the  {70^(7),  N  <  oo}  is  uniformly 
integrable,  and  so  is  {f/*'^(n  +  1)  —  t/‘’^(n),  n  <  oo,  N  <  oo,  »  =  1,2}  by 
Theorem  13.  Since  T]\-  }  oo,  we  can  suppose  that 

<(•))  (Xo(  ).  Jo(  ),  Wo{  )) 
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and  reflection  processes.  The  set 


{A'^p(  ),  JAp(  ),H'(  ).t^Ap(  ).W  ).  A  >  0,p  >  0}  (6,5) 

is  tight  and  the  weak  limits  all  satisfy  (3.12),  (3.13).  By  the  uniqueness  (A6.2), 
the  limit  of  any  weakly  con%'ergent  subsequence  of  the  set  (6.5)  satisfies  (3.12), 
(3.13).  Then  (6.4)  follows  from  the  weak  convergence  and  the  boundedness  of 
7(  )  and  the  consequent  uniform  integrability  of  {U'^^{n+  1)  -  n  <  oo, 

A  >  0,  p  >  0,  i  =  1.2}.  Q.E.D. 

For  A  >  0.  p  >  0,  let  denote  the  set  of  admissible  (with  respect  to  some 
given  Wiener  process  U'(  ))  controls  which  are  bounded,  are  constant  on  each 
interval  [nA,  riA  +  A),  jump  only  at  the  times  nA,  and  J‘-’(nA)  —  J*-’  (nA“)  is 
an  integral  multiple  of  p.  By  Theorem  14  and  (A6.2),  we  know  that  for  each 
^  >  0  there  are  A  >  0.  p  >  0  such  that  there  is  a  6-optimal  control  in  some 
'T^p-  We  will  need  to  define  this  control  in  such  a  way  that  it  can  be  used  for 
the  physical  A'‘(  )  process. 

Write  k  =  {ki.  k2).  a  multi-index,  where  k,  is  either  an  integer  or  0.  Fix  the 
Wiener  process  It  (•)  and  A  and  p.  For  J(  )  €  (J(  ),U’(  ))  is  admissible. 

For  1  >  0  and  integers  k  and  n,  define  qnki{-)  by 

9nt-r(>/(mA),m  <  n,W{£y),(y  <  nA) 

=  F’{dJ(nA)  =  Fpl  J(mA),m  <  n,  W(f7),f7  <  nA).  (6.6) 
By  the  martingale  convergence  theorem,  as  7  — »  0 
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qnk-,U{mA).m  <  n,W((y),ey  <  nA)  — 

P{dJ(nA)  =  kp  I  J(mA),  m  <  n,  W'(s),  s  <  nA} 

w.p.l  (\\’iener  measure)  for  each  k,n  and  value  of  {J(mA),m  <  n}. 

For  each  y  >  0,  we  next  choose  a  control  7,(  )  €  recursively  by  means 
of  the  following  set  of  conditional  probabilities. 

P{dJ-,{riA)  =  kp  I  J.y(mA),  m  <  n.  U‘(s),  s  <  nA)  = 

=  q„k-,{J-,(mA),m  <  n,W[(y),ty  <  nA).  (6.7) 

(6.7)  specifies  the  joint  law  of  admissible  W’(  )),  and  there  is  a  solution 

A'(  )  on  some  sample  space  which  is  associated  with  a  pair  with  the  same  dis¬ 
tribution  as  (J-,(  ),  IF(  )). 

We  will  need  the  following  condition. 

A6.3.  The  uncontrolled  system  (J{t)  =  0)  has  a  unique  (tn  the  weak  sense) 
solution  for  each  intial  condition. 

Theorem  16.  Assume  (A6.1)-(A6.3).  Let  (J(  ),  fV'(  ))  be  admissible  with 
J(  )  €  fo'c  some  A  >  0.  p  >  0.  Define  Jn,{  )  as  above.  Then 

V{x,Jy,W)-*Vix,J,W).  (6.8) 

The  function  ?„*,(  J.,(mA),  m  <  n,  •)  which  is  used  to  get  Jy{  )  can  be  chosen 
to  be  continuous  for  each  n,  k,  y,  and  values  of  the  set  {Jy{mA),  m  <  n}. 

Proof.  The  prcxjf  of  (6.8)  follows  from  the  weak  convergence  {7.y(  ),  1V(  ), 
■y  >  0}  =>  (J(  ),  H’(  )),  as  7  — •  0  and  the  uniform  boundedness  of  the  controls. 
By  (A6.3),  the  solution  to  (3.12),  (3.13)  is  unique  in  the  weak  sense  for  any 
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admissible  J{  )  in  T^p.  The  last  sentence  of  the  theorem  follows  from  the  fact 
that  for  each  n,  k,  7,  and  value  of  {J^(mA),m  <  A},  we  can  approximate 
<  ”r)  by  a  sequence  of  continuous  distribution  functions, 
which  converge  to  q„ic-,{Jy(mA),  m  <  n,  •)  w.p.l  (Wiener  measure).  Q.E.D. 

The  optimality  theorem.  We  now  return  to  the  physical  process  (2.8), 
and  prove  equality  in  (5.8).  Let  (Jy{  ),W{  ))  be  admissible  with  ./.,(•)  chosen 
by  (6.7),  where  the  qnkyi  )  have  the  continuity  property  asserted  in  Theorem 
15.  Recall  the  definition  of  W'(  )  given  above  Theorem  6. 

We  would  like  to  define  a  control  J'(-)  for  X'(  )  such  that  {d'(  ),  1L''(  )} 
converges  weakly  to  (J,(  ),W'(  ))  as  f  — ►  0.  First,  consider  the  control  j'(  ) 
defined  as  follows,  where  the  qnkyi  )  are  continuous  in  the  u>-arguments;  J‘{  ) 
is  constant  on  the  intervals  [nA,  nA  +  A)  and 

P{clJ‘{nA}  =  kp  I  J'(mA),  m  <  n,  H’'(s),  s  <  nA} 

=  P{u7‘(nA)  =  |/'(mA),m  <  n,  W'(»')),  »7  <  nA)  (6.9) 

=  W''(»7).  »7  <  ”A). 

The  control  law  (6.9)  can’t  quite  be  realized  for  A’'(  ),  since  the  controls  for  A''(  ) 
are  the  result  of  rerouting  decisions  and  A''(  )  cannot  be  impulsively  controlled. 
But  we  can  come  close  enough  to  realizing  the  above  /'(•),  as  follows. 

For  notational  simplicity,  let  the  F.p,  »  =  1,2,  be  integral  multiples  of  -y/e. 
Let  A<  -*  0  as  <  —  0  such  that  AJy/i  00.  Let  denote  the  event  that 
there  are  >  (Bj  +  arrivals  at  Pq  on  [nA,nA  +  Ae).  WehaveP{g7}  —  1 

as  c  — ►  0.  Define  y‘(  )  to  be  any  control  with  the  following  properties:  y'(  ) 
is  constant  on  [nA  +  A(,  nA  +  A)  and  on  [0,A);  for  n  >  0,  the  rerouting 
[nA,nA  +  A<)  is  such  that 
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P{J‘(nA  + A,)  -  J‘(nA)  =  i:/)lJ‘(mA),m  <  n,W'(s),s  <  nA.g^} 


=  P{J'(nA  +  Aj)  -  J'(nA)  =  kp\J‘{mA),m  <  n,W‘{iy),iy  <  nA,Q7} 

=  9ntT(*/*(*^A),m  <  n,  t7  <  nA).  (6.10) 

The  limit  of  the  costs  associated  with  this  just  constructed  J'(-)  is  the  same 
as  if  the  jumps  of  J'(  )  are  at  the  time  nA  +  Aj,n  =  1,2, ,  only  and  not 
spreadout  over  [nA,nA  +  A^).  This  can  be  easily  proved  by  the  “time  charge 
method".  Under  this  “new”  J'(  ),  { J'(  ),  W'(  )}  clearly  converges  weakly  to 
( U'(  ))  as  (  — >  0.  Now,  the  above  discussion  and  Theorems  6,  10  and  11 
yield  ihe  following  theorem,  where  we  use  the  J‘{  )  just  described.  Note  that 
the  rescalings  are  not  necessary,  due  to  the  fixed  form  of  J'(  );  we  get  weak 
convergence  directly  in  the  Skorohod  topology. 

Theorem  17.  Assume  (A2.1)-(A2.5)  and  (A6.1)-(A6.3).  Then  {A'‘(  ), 
V‘(  ).  (•'(■),  U''(  ).  7‘(-)}  converges  weakly  to  (A'(  ),  y(  ),  U{  ),  Wi  ),  7(  )), 
where  IU(  )  =  (Uai(  ).  Udi(  )'  *  =  0,1,2)  and  ihe  limit  satisfies  (3.12),  (3.13). 
Also 

VHx,r)^V{x.J^,W),  (6.11) 

U'(x)-V(x).  (6.12) 

Remarks.  In  [4],  it  was  shown  that  certain  forms  of  nearly  optimal  controls  for 
the  limit  process  were  also  nearly  optimal  for  the  physical  process  under  heavy 
traffic.  A  similar  situation  bolds  here,  2Uid  this  partly  justifies  the  use  of  the 
heavy  traffic  limit,  but  we  reserve  the  comments  for  a  future  paper  on  numerical 
methods. 
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Suppose  that  ki  >  0  but  k2  <  0  and  |Ar2|  <  'ti-  Then  a  very  similar  analysis 
can  be  carried  out,  with  similar  results.  The  costs  V‘(x)  can  be  bounded  from 
below  since  the  profit  to  be  made  by  rerouting  from  Pi  to  P2  is  bounded,  due 
to  the  limited  idle  time  at  P^. 
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7.  A  More  General  Network  Model 


The  general  controlled  routing  open  network  version  of  Figure  1  can  be 
treated,  for  any  number  of  servers.  Because  of  the  notational  burden  involved 
in  writing  all  the  possible  ‘rerouting  terms’,  we  give  the  extension  only  to  the 
model  of  Figure  2,  which  differs  from  Figure  1  only  in  that  feedback  is  allowed. 
We  continue  to  use  the  notation  of  the  previous  sections,  except  for  the  following 
additions.  Let  be  the  indicator  of  the  event  that  a  service  completed  at  P, 
(i  ^  0)  at  real  time  n  is  routed  to  Pj  {j  ^  i)  if  j  ^  0,  and  leaves  the  network  if 
j  =  0.  The  in,  ut  from  Pj  to  Pi  is 

*/« 

=  n/7  j  =  1,2,  .■  =  0,1,2. 

m=l 

denotes  the  scaled  number  of  outputs  of  Pj  which  leave  the  system 
directly.  The  ‘fictitious'  outputs  from  Pj  (which  are  due  to  our  convention  of 
Pj  ‘processing'  even  with  an  empty  queue)  and  which  are  sent  to  P,  are 

«/« 

m=:l 

The  overflow  due  to  a  full  buffer  at  P,  is 
«/< 

V'  V)  = 

m=:l 

Then,  for  j  ^  i,  j  ^  0, 

-  +  (7.1) 

We  continue  to  use  the  cost  functional  (2.3). 

Replsice  (A2.3)  and  (A2.4)  by 


37 


A2.3'.  There  are  <  1  such  that  {p^i  replaces  the  p^  of  (A2.3)) 

=  1 1  a//  arrival  and  service  intervals  and  routings  starting  by 


time  n,  except  for  I^'^}  = 

A2.4'.  gji  =  gai+Poigao  +Pji9dj,  j  /  ».  ij  #  0. 

In  analogy  to  the  definitions  of  the  centered  processes  v4o‘(  ),  '(')  given 

in  (2.5),  define  (j  ^  0) 

‘/'  _ 

Ef4‘i  „  - 

m=l 

Define  the  centered  reflection  process 

«/< 

m=l 

Define  £>'•’(0  =  (t)).  By  the  same  method  which  was  used  to  get  (2.8), 

we  can  write  (7.1)  in  the  form  (»  ^0,  j  gt  0,  j  ^  i) 

A"  ‘(0  =  A'’''(0)  +  fl’’'(0  +  i’'*(<)  +  A'°‘‘(t)  -  D‘°'^it)  -  D'^'‘{t) 

+  ^^’'(o + y''(o  -  pjiy^’Ht)  -  -  u'-^it) + [/>•’'(<)  -  r^^io] + (5’’'(o, 

(7.2) 

where  ^'''(■)  is  as  in  (2.7). 

Theorem  17.  Assume  (A2.1),  (A2.2),  (A2.3'),  (A2.4'),  (A2.5).  Then  the 
five  sets  of  processes  (we  pair  as  (D'°,  D**),  (D^°,Z?*')) 


{A'''(),i^‘(),(A°‘''(),A°2'()),(D‘°-'().D’=*’'()),(P^°X).^'‘''()).f>0} 


are  tight.  The  limits  are  continuous  martingales.  All  of  Theorems  2  to  16  hold, 
with  the  obvious  changes  necessitated  by  the  additional  (erms  in  (7.3).  The  limit 
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reflected  diffusion  is 


X'(t)  =  X'(0)  +  A‘(t)  +  -  D‘°(t)  -  D‘^(t)  +  D^‘(t)  -  B‘(i) 


+  J‘(t)  -  [/‘(i)  +  V*(t)  -  PjiY‘(i),  i  ^  j.  (7.4) 


Also,  for  i  ^  j 


quad  var 


where 


Y'dij  (2")  —  gdi 


Pio(l  ~  Pio) 
-PioPij 


~Pi0Pij 
Piji^-Pij)  . 


1 

PioPij 


(7.5) 


Proof.  All  the  details  are  copies  of  what  was  done  in  Theorems  2  to  16, 
except  for  the  treatment  of  the  y*^’‘(  )  term,  and  some  details  in  Theorem 
7.  Define  >’'•'  '(<)  =  y'-’'‘(r'(<)).  The  summands  in  the  y'^’'(  )  are  centered 
about  their  conditional  expectations,  given  the  ‘past’,  and  hence  y'^''(  )  is  a 
martingale  sum.  Its  variance  is  bounded  by 

E  hxi,^=o)  =  0(t).  (7.6) 

msO 

Since  the  summands  (without  the  y/e  included)  in  y'''’'(  )  are  uniformly  square 
integrable,  {y‘-’'‘(  ),  f  >  0}  is  tight,  and  all  weak  limits  are  continuous  processes. 
Write  the  scaled  form  of  (7.2)  as 


A’’'(t)  =  A-j  '  +  Z*>(t)  +  [y’-‘  -  PjiY’  ^t)]  -  {/•■'(<)  (7.7) 

with  the  obvious  definition  of  2‘''(  )-  {Z*‘‘(  ),€  >  0}  is  tight  and  all  weak  limits 
are  continuous  processes.  Thus,  by  Theorem  1,  {y*’'(  ),  ^''‘(•)>  e  >  0}  is  tight 
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and  all  weak  limits  are  continuous  processes.  This  implies  that  for  each  t  <  oo, 

msO 

is  bounded  in  probability  uniformly  in  f,  for  otherwise  some  subsequence  of 
y’'(t)  would  go  to  infinity  with  a  positive  probability.  Hence  the  left  side  of 
(7.6)  goes  to  zero  as  €  —»  0  for  each  t  <  oo.  Thus  )  =>  zero  process. 

Theorem  7  also  continues  to  hold,  since  in  the  present  case  we  write  (4.1)  as 

[t’’''(n  +  1)  +  y2^'‘(n  +  1)]  -  [t/^-‘(n)  +  y2^''(n)]  =  right  side  of  (4.1). 

The  left  hand  side  of  (4.3)  now  becomes 

p{  sup  [(B*  '(s)  +  Afi''(s)-y2i-'(s))-(B'''(r)  +  A/'''(r)-y2i.'(r))]  >  5o  | 

data  up  to  t}. 

Since  y^'  '(■)  is  non-decreasing,  the  expression  is  still  <  1  -  oo  for  small  enough 
^0,  and  we  can  continue  the  proof  of  Theorem  7.  Q.E.D. 
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8.  Proof  of  Theorem  1 


For  notational  simplicity,  we  prove  the  theorem  for  F  =  2  and  then  comment 
on  the  general  case.  The  proof  in  the  general  case  is  the  same  in  all  essentials. 
The  following  result  is  proved  in  [1],  [2]. 

Lemma.  Let  P  be  a  degenerate  Markov  transition  matrix  whose  spectral 
radius  is  less  than  unity.  Then  there  is  a  unique  ‘non-anticipative’ function  F{-) 
with  the  following  properties:  F{  )  maps  £)*[0,oo)  into  D*[0,oo)  and  C*[0,oo) 
into  C*[0  , oo),  and  is  continuous  in  the  topology  of  uniform  convergence  on 
bounded  intervals.  Let  i(  )  G  D*[0,oo)  and  define  y(  )  =  (y'(  ),  »  <  t)  = 
F(2(  )).  Define  x{t)  =  !(<)  +  (/  —  P')y(t).  The  j/*(  )  are  non- decreasing  and 
y‘(  )  can  increase  only  when  the  x'(0  =0-  *‘(0  ^ 


To  prove  Theorem  1,  we  will  use  the  lemma  in  a  ‘sequential’  way.  Refer  to 
Figure  3.  We  can  assume  w.l.o.g.,  that  the  diagonal  entries  in  Q  in  (3.1)  are 
zero.  There  are  four  different  reflection  maps  which  appear  in  (3.1),  depending 
on  which  segment  of  the  boundary  is  involved.  On  the  boundary  (d, o,6)  = 
segment  1,  our  system  (3.1)  is 


x{t)  =  z(t)  + 


1  —921 

~9l2  1 


y(t),  i’(<)  >  0. 


(8.1) 


For  the  system  (8.1),  with  the  constraint  x*{i)  >  0,  the  lemma  defines  a  contin¬ 
uous  mapping  z{  )  — ►  (y(  ), «(•)),  where  «(•)  =  0.  Call  this  mapping  Fi{  ).  On 
the  other  segments,  the  system  is 

x(t)  =  z(t)  -h  J  Q  y(t)  -  .  segment  2  =  (a,b,e),  (8.2) 


T(t)  =  z(t) 


_  f  "‘(0  \ 

I  /  ’ 


segment  3  =  (6,  c,  d) 


(8.3) 
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x(t)  =  z(t)  +  Q  y(<)  -  ^  ^  .  segment  4  =  (c,d,a).  (8.4) 

The  reflection  maps  for  (8.2)  to  (8.4)  (with  the  sides  extended  to  oo)  are 
trivial,  as  we  will  now  see,  since  they  are  each  just  concatenations  of  two  one 
dimensional  applications  of  the  lemma.  Let  F2(')  denote  the  map  associated 
with  (8.2)  which  sends  2(  )  into  (y(  ),  «(•))•  ^2(  )  i®  constructed  as  follows. 
First,  we  have  y^(  )  =  u^(  )  =  0.  Then  y^(  )  is  defined  by  the  lermna  for  it  =  1; 
in  particular  y^(f)  =  —  min{0,inf,<j  z^(s)}.  Finally,  the  t/^(  )  is  defined  by  the 
reflection  needed  to  keep  x^(t)  <  Bi;  i.e.,  0  <  Bi  —  a:^(<)  or 

(8.5) 

Similarly,  we  define  the  analogous  continuous  map  F3(  )  and  F4(  )  associated 
with  (8.3)  and  (8.4)  resp.  [The  calculation  of  the  y(  )  and  u(  )  always  decouples 
into  two  separate  calculations;  first  getting  y(  )  and  then  getting  u(  ),  even  for 
the  general  it  case  as  seen  below.] 

Define  5  =  [0,Bi]  x  [0,52]'  Let  2(0)  €  S  w.l.o.g.  Define  x(-),  yn(  ),  u„(  ), 
in(  ),  iyn(  ),  ^«n(  ),  and  r„  as  follows:  Tq  =  0,  2o(  )  =  2(  ),  6uo(  )  =  ^yo(  )  =  0, 
Ti  =  inf{<  :  2(<)  ^  5},  u(<)  =  y(t)  =  0  on  [0,ri]  and  x(t)  =  *o(0  =  «(<)  on 
[0,ri].  In  general,  for  n  >  1,  given  r„,  Xn-i(  )  and  y(  )  and  u(  )  on  [0,r„],  define 

=  ^(0  +  [f  -  Q']y(t  n  r„)  -  u(t  n  r„), 
s„  =  a  boundary  segment  (1,2,3  or  4)  on  which  Xn_i(r„)  lies, 

(^Un('),^yn(-))  =  ^ (8'6) 
Xn{t)  =  2n(t)  +  [f  -  Ql^yfi(0  - 
x(<)  =  x„(0  for  t<T„+i, 


u^(<)  =  -  min  |o,  inf(Bi  -  2^(s)  + 
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T„+j  =  inf{t  ;  x„(t)  0  S}, 
u(<)  =  u(r„)  +  iu„(<),  for  t  €  [t„,  t„+i], 
y(<)  =  y(T-n)  +  ^yn(0.  for  t  e  [tv,,  tv,+i]. 

Note  that  2n(0  €  S  until  at  least  time  t„  and  *n(0  G  S  until  at  least 
time  r„+i.  Hence  (6u„(<),  6j/n(0)  =  ^  rv,.  Also  Xn{t)  =  ^n(0> 

<  <  r„,  2„+i(<)  =  2„(t)  on  [0,r„+,],  x„+i(t)  =  *„(<)  on  [0,tv,+i].  The  idea  in 
constructing  Zn(  )  is  that  when  Xn(  )  exits  S  on  a  certain  boundary  segment, 
we  use  the  reflection  map  (Fi,  Fj,  F3  or  F4,  as  appropriate)  for  that  segment 
to  get  until  Xn4.i(  )  exits  S.  It  must  exit  on  a  ‘different’  segment. 

F,,(  )  can  be  any  map  associated  with  the  boundary  segment  on  which  the 
exit  point  x„_i(r„)  lies.  Except  for  in  the  corner  points,  there  are  two  such 
maps  associated  with  each  point  on  the  boundary.  Which  map  is  chosen  is 
immaterial.  For  definiteness,  choose  Fi(  )  on  [e,o,/],  F2(  )  on  (/, i.y],  Fai-)  on 
(g,c,h]  and  F4(  )  on  (h,d,e).  We  can  verify  that  r„  -*  00  and  (by  induction 
using  the  lemma)  that  the  constructed  x(  ),  j/(  ),  u(  )  satisfy  Theorem  1.  To  see 
that  the  choice  of  the  map  which  is  used  at  the  points  /,  g,  h,  e  is  immaterial, 
let  x„_](t„)  =  /.  Then  F(2„(  ))  =  F2(2n(  ))  until  the  inflmum  of  the  times 
that  x„(  )  leaves  S  through  [6,c,d,a].  An  induction  argument,  based  on  this 
observation  shows  that  the  choices  at  the  points  /,  g,  h,  e,  is  immaterial. 

Remark  on  the  general  k  >2  case.  There  is  always  a  decomposition  of 
the  construction  of  (6y„(  ),  ^u„(  ))  into  the  two  sequential  steps:  first  calculate 
^Vni  ’)  via  the  lemma,  for  a  reduced  system;  then  calculate  the  ju|^(  )  individually 
via  an  appropriate  analog  of  (8.5).  Just  to  illustrate  this  point,  consider  the 
case  where  the  space  is  5  =  and  focus  on  the  faces  of  S  meeting  in 
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the  corner  (B,  B,  0, 0, . . .  ,0).  On  these  faces  (excluding  the  edges  which  do  not 
touch  (B,  B,0,0, . . .  ,0)),  (3.1)  is 


x(<)  =  z{t)  + 

0  0  “931 

0  0  “932 

■■  -9ii  ' 
■  •  -9*2 

y(0“ 

f  «'(<)  \ 

«2(t) 

0 

o 

1 

- 

<  0  / 

where  7  —  Q'  is  a  reduced  transition  matrix.  Then  first  get  (y^(  ),  ..  .,£/*(■)) 
from  the  lemma,  and  then  define  (for  i  =  1,2) 


u’(0  =  -  min  f  0, inf (B-  2^(0  +  ^gjiSr'^)) 

I  *-  3=3 
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Figure  1. 
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